Computer Vision Algorithms for Yield Mapping in Specialty Farms

University of Minnesota Ph.D. dissertation. November 2019. Major: Computer Science. Advisor: Volkan Isler. 1 computer file (PDF); xii, 175 pages.Precision farming and phenotyping technologies have the potential to drastically transform the agricultural landscape. For commodity crops such as maize, wheat and soy recurring farming tasks such as seeding, weeding, irrigation, fertilization, application of pesticides, harvesting, and storage are in the process of being completely automated. Specialty crops (tree fruit, flowers, vegetables, and nuts) are excellent candidates for similar automation as they have high monetary value, high management cost and high variability in growth. An important capability for both precision agriculture and phenotyping is yield mapping. Yield mapping for tree fruit is challenging because it involves solving multiple computer vision (fruit detection, counting, recovering underlying 3D geometry for tracking fruit across different frames in continuously changing illumination) as well as planning problems (path planning for covering all fruit, picking fruit). The main goal of this dissertation is to develop computer vision and deep learning algorithms for yield mapping in specialty farms. The dissertation is divided into three parts. The first part is dedicated to developing practical solutions for yield mapping in specialty farms. We present solutions for fruit detection, counting, recovering the underlying scene geometry and fruit tracking. We integrate these individual solutions in a modular manner and create a flexible framework for complete yield estimation. Additionally, we perform an extensive experimental evaluation of the developed system and sub-components. Our algorithms successfully predict 97% of the ground truth yield and outperform all existing state-of-the-art methods. Some of these efforts are now in the process of being commercialized. In the second part of the dissertation, we study a problem where a manipulator equipped with a camera, mounted on a ground robot is charged with accurately counting fruit by taking a minimum number of views. We present a method for efficiently enumerating combinatorially distinct world models most likely to generate the captured views. These are incorporated into single and multi-step planners for accurate fruit counting. We evaluate these planners in simulation as well as with experiments on a real robot. In the third part, we study the problem of realistic synthetic data generation for training deep neural networks. We present a method that jointly translates the synthetic images and their underlying semantics to the domain of the real data so that an adversarial discriminator (a deep neural network) cannot distinguish between the real and synthetic data. This method enables us to stylize the synthetic data to any fruit, lighting condition and environment. It can be applied to a wide variety of domain transfer tasks beyond fruit detection and counting (e.g from Grand Theft Auto (GTA) to Cityscapes for autonomous driving). Additionally, it enables us to perform image to image translation with significant changes in underlying geometry (e.g circles to triangles, sheep to giraffe, etc). These results in this dissertation together present a complete yield monitoring system for specialty crops, view planning strategies for accurate fruit counting and a framework for generating realistic synthetic data. These methods together push the state-of-the-art and take us one step closer toward building a sustainable infrastructure for intelligent integrated farm management

Registering Reconstructions of the Two Sides of Fruit Tree Rows

We consider the problem of building accurate three dimensional (3D) reconstructions of orchard rows. This problem arises in many applications including yield mapping and measuring traits (e.g. trunk diameters) for phenotyping. While 3D reconstructions of side views can be obtained using standard methods, merging the two side-views is difficult due to the lack of overlap between the two partial reconstructions. We present a novel method that utilizes global features to constrain the solution. Specifically, we use information from the silhouettes and the ground plane for alignment. The method is evaluated using multiplesimulated and real datasets

MinneApple: A Benchmark Dataset for Apple Detection and Segmentation

The counting and detection data are used to solve two different subproblems of the larger problem that is yield estimation. Yield estimation relies on accurate detection of the fruit. Since fruit can be clustered together, it is necessary to use a separate algorithm for counting (in most cases). We provide data to develop and test algorithms for both subproblems.We present a new dataset with the goal of advancing the state-of-the-art in fruit detection, segmentation, and counting in orchard environments. We hope to achieve this by providing a large variety of high-resolution images acquired in orchards, together with human annotations of the fruits on the trees. Objects are labeled using polygon masks for each object instance to aid in precise object detection, localization or segmentation. Additionally, we provide data for patch-based counting of clustered fruits. Our dataset contains over 40'000 annotated object instances in 1000 images.USDA NIFA MIN-98-G0

Robotic Surveying of Apple Orchards

We present a novel system for surveying apple orchards by counting apples and estimating apple diameters. Existing surveying systems resort to active sensors, or high-resolution close-up images under controlled lighting conditions. The main novelty of our system is the use of a traditional low resolution stereo-system mounted on a small aerial vehicle. Vision processing in this set up is challenging because apples occupy a small number of pixels and are often occluded by either leaves or other apples. After presenting the system setup and our view-planning methodology, we present a method to match and combine multiple views of each apple to circumvent these challenges and report results from field trials. We conclude the paper with an experimental analysis of the diameter estimation error